DOCUMENT RESUME 



ED 075 086 



PS 006 421 



AUTHOR 
TITLE 

INSTITUTION 
SPONS ASENCY 

REPORT NO 
PUB DATE 
NOTE 



EDRS PRICE 
DESCRIPTORS 



Alkin, Marvin C. ; Kosecof £ , Jacqueline B. 
A New Eclectic Model for the Redirection of 
Evaluation Efforts . 

California Univ., Los Angeles. Center for the Study 
of Evaluation. 

Office of Education (DHEW), Washington, D.c. Research 
and Development Centers Branch. 
CSE-R-84 
Mar 73 

13p. ; Paper presented at the Annual Meeting of the 
American Educational Research Association (New 
Orleans, Louisiana , February 25 - March 1, 1973) 

MP-J0.65 HC-S3.29 

^Counseling Services; *Decision Making; ^Evaluation 
Techniques ; *Models; *Motivation ; Objectives ; 
Political Influences; Social Influences; Speeches 



ABSTRACT 

A discussion of client evaluation is presented One 
of the initial elements in performing an evaluation is "framing the 
decision context. 11 In framing this context, the evaluator is faced 
with several key questions, for example, "Who are the decision makers 
or decision audiences to whom he will be presenting information?" and 
"What are the likely decisions that will be made as a consequence of 
information that he might present?" Additional dimensions of the 
decision context are explicit decision-making purposes, implicit 
decision-maker motives, developmental stage of programs, ?nd the 
program's socio-political setting. (CK) 
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This presentation is billed as "A New Eclectic Model of Evaluation." In 
all honesty, we must admit to you that the "new" part was not included in our 
original proposal but seems rather to have been added along the way. Evaluation, 
or for that matter most social science disciplines, is simply not prone to 
producing totally new ideas, for in the "soft science" areas there is always some- 
one who has alluded to at least the germ of an idea, that we delude ourselves into 
thinking of as new. (This situation is epitomized by the frequently posed query: 
How old must an idea be before it is new?). For instance, long before evaluation 
had matured into a distinct discipline, Alvin Gouldner (1961) suggested a continuum 
of alternate approaches to consulting that closely parallels current "new" 
approaches to evaluation efforts. The basic assumption underlying this continuum, 
termed the "value free" postulate, is that social science cannot formulate and 
specify objectives for the client group. At one end of this continuum the 
consulting "engineer" conceives and conducts his assignment largely in terms 
formulated by his client. In contrast, the consulting "clinician" is less 
likely to take his client's own values as given and instead derives his own diagnosis 
based only in small part upon the client's complaints and self- formulations. 

The engineering approach has been exemplified most nearly in recent 
educational evaluation by the writings of Stufflebeam and associates (1971) , 
Provus (1971), Popham (1972) and Alkin (1969, 1971) while the clinical approach 
has been represented most nearly by the writings of Scriven (1971) , Stake (1967) 
and Glass (1969). The Gouldner notion of clinical evaluators, who are allowed 
to adopt a "value free" stance in the way in which they perform their evaluations, 
has its modern-day counterpart in the "goal free" pronouncements of Scriven (1972). 
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This lengthy set of ramblings is intended simply to point out that possibly there 
are far fewer M new ,T ideas than we would like to believe and that our emphasis 
might well be in concentrating on putting together M old but still good ideas" in 
eclectic or unique ways, (Having justified the title of our paper, we may proceed 
with saying what we really wanted to say.) 

We start our discussion of evaluation with cur philosophical feet squarely 
placed toward the engineering end of Gouldner's value free continuum. Thus, we 
see a distinction between evaluation on the one hand and general research on the 
other; we see a distinction between evaluation attentive to a decision maker's 
information needs on the one hand and evaluation which establishes its o\vn 
information priorities and agenda on the other. And, at this end. of the continuum s 
we are committed to working with the clien t to define a representative and realistic 
problem statement. 

Framing the Decision Context 
Given this perspective we have identified one of the critical, initial 
elements in performing an evaluation as "framing the decision context/' That 
is, in our view one of the most important steps in the evaluation process occurs 
prior to the kinds of activities that most people think of as evaluation. In 
this step, the framework in which the rest of the evaluation is to take place is 
established. Thus, when framing the decision context, the evaluator is faced 
with several key questions : Who are the decision makers or decision audiences 
to whom he will be presenting information? What are the likely decisions that 
will be made as a consequence of information that he might present? What are 
some of the socio-political factors in this situation that will delimit or modify 
the kind of evaluation that he might perform? And, in what stage of development 
is the program that he has been called upon to evaluate? 
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For purposes of this discussion, and with innumerable caveats about the 
enormity of the task and the time constraints of this presentation, let us 
consider five elements, of the decision context, as follows : (1) the decision 
maker(s) and/or decision audiences; (2) the explicit decision-making purposes; 
(3) the implicit decision-maker motives; (4) the developmental stage of the 
program to be evaluated; and (5) the socio-political setting. While these five 
elements are not an entire evaluation model per se (nor are they intended to be) , 
they nonetheless jointly constitute, in our view, the most important step of the 
evaluation task. 

The Decision Maker (s) and Decision Audiences 

Identification of the program's decision maker (s) is perhaps the most 
elusive variable associated with a decision context. Different organizations 
characteristically have very different decision structures ranging from those in 
which specific decision responsibility is emphasized to those in which rather 
amorphous divisions of decision-making responsibility exists. The spectrum of 
potential decision makers is equally diffuse, ranging from those individuals who 
have daily contact with the target program to those more distant from the program 
but who have, for example, provided for its funding. Thus, evaluative information 
might be used by program staff (teachers), the program director, other program 
administrators, district administrators, community groups, special interest groups, 
and program sponsors (state, federal or private granting agency) or Congress. 

One of the major problems for the evaluator is identifying the decision 
audience to whom his reporting will be primarily directed. We would maintain that 
a single evaluator (or evaluation team) cannot adequately serve two decision 
makers or audiences simultaneously. "No man can serve two masters, for. either 
he will hate the one and love the other, or else he will stand by the one and 
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despise the other" (Matthew: 6:24). The important thing is, however, that the 
evaluator be quite clear as to which decision maker or audience he has responsibility 
for evaluation information reporting. General purpose information provision cannot 
be considered evaluation. 

Explicit Decision-Mak in g Purposes 

A variety of uotential decision makers implies that there will be a 
potential variety of decision-making purposes. Even if a particular decision 
maker is identified, there are many possible kinds of decisions that might be 
contemplated. Some of the decisions that might be anticipated include program 
change and modification as a result of the evaluation, refunding, personnel 
changes, program adoption, or program generalization. Careful examination and 
an attempt to make explicit the avowed decision purposes is an essential element 
in framing the decision context. 

The issue of evaluator- decision maker responsibility and conflicting 
decision purposes needs clarification before an evaluator can proceed very far 
in his task. For example, consider the ambi<~ iity that presently exists for an- 
evaluator engaged in the evaluation of a Title VII bilingual project in a school 
district. First, let us assume that the evaluator was selected by the individual 
Title VII project director. One of his prime responsibilities is for the 
provision of formative evaluation data to that local project director to assist 
him in making program modifications. Second, there is a requirement within the 
project for a "final report." Several potential decisions will emanate from 
such a report. On the one hand, the district superintendent or assistant super- 
intendent will review the report in order to make judgments about the quality 
of the program for potential implementation elsewhere in the district. A hidden 
agenda item will involve making judgments about the effectiveness of the project 
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director. The report on the outcome of the program will also be sent to the 
Title VII program monitors at the U.S. Office of Education for their review. 
Presumably the decision question that they have in mind would be something like, 
"Is this project working and should funding be increased, renewed, decreased, 
or terminated? n Now, consider the dilemma of the poor evaluator who attempts to 
provide direct information for these three decision needs simultaneously. 

We are not suggesting that framing the decision context, and in particular 
being aware of the decision makers for whom information is to be provided and the 
particular decision needs of each, will solve the plight of the Title VII evaluator. 
(The dilemma faced by these poor souls seems to place them in a position 
beyond redemption.) What is instead suggested is that if the evaluator will 
initially identify the decision makers of concern in his evaluation and seek 
clarification of his role and function vis-a-vis each of them, he will save 
himself a great deal of effort, produce more relevant findings, and have a better 
basis for justifying the soundness of his work. 

Implicit Decision-Maker Motives 
^ ^ A third dimension of decision-making context, which is related to is explicit 
< f purpose, is the implicit motives of the decision maker for commissioning the 

^valuation. The distinction between the explicit and implicit -dimensions of the 
^v4ecision-maker activities is not absolute but rather is based on how the decision 
>'^aker intends to use the evaluative information. Explicit decisions, such as those 
,, ^discussed in the previous section, are .generally characterized by well-defined 
PlLf eCisi0n areaS rec l uirin g vei 7 specific kinds of information. Illicit motives, 
on the other hand, frequently involve the commissioning of an evaluation and its 
subsequent performance as a pro forma exercise ; the act of having conducted an 
evaluation is the crucial element, and the specific evaluation information is most 
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frequently irrelevant. Implicit decisions can be quite important and should not 
be viewed as peripheral or illegitimate for lack of a formal decision purpose. 
Thus, some implicit decision -maker motives might be (1) the satisifaction of legal 
requirements -- "The legislation says we've got to do an evaluation, so do it but 
don't bug us too much"; (2) justification of the program -- "We know our program 
is superb and simply want to have an evaluator document this M ; (3) public 
relations "We are a lighthouse school district and since evaluation is popular 
we would like our community to know that we did one"; (4) professional prestige 
"I am an administrator on the move and it would help my image as a progressive 
educator to have commissioned an evaluation"; (5) community pressures or pressures 
from within the school system "An evaluation will help to get them off our 
backs." 

This is a most disheartening list of raisons d'etre for an evaluation. Yet, 
the list is believable by all but those naive to the field. 

Developmental Stage of Progr ams 

Another important element to be considered in framing the decision context 
is the developmental stage of the program. Just as individuals in their life 
cycles go through developmental stages, so too must we consider an education 
program as a living, growing entity that proceeds through various developmental 
stages. During a program's childhood, or the planning stage, program developers 
must define those goals and objectives they wish, to address and select the most 
appropriate program to meet these ends. A program's adolescence, or formative 
years, typically requires close parental supervision in order to assure that 
the desired program has been implemented as planned and is making satisfactory 
progress. Any problems that arise during this developmental stage should be 
immediately treated by introducing needed modifications and changes. In this 
fashion a program can improve as it matures. Finally upon reaching adulthood 
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(the completed product stage], a program's outcome and impact will be judged by 
society. 

Thus, we conceive of three basic developmental stages of programs: the 
planning stage, the formative stage, and the product stage. In turn, each stage 
consists of two substages as illustrated in Table 1. 

Table 1: Stages of Program Development 



Planning Stage (Childhood) 
Goal Definition 
Program Selection 

Formative Stage (Adolescence) 

Program Implementation 
Monitoring Progress 

Completed Product (Adulthood) 
Outcome 
Impact 



An evaluator should be aware of two characteristics of these developmental 
stages of educational programs. (1) Developmental stages whil? 11 time- related 1 1 
are not u time-bound. M Thus, while there is a sequence of program developmental 
stages, it is not guaranteed that all programs will attain these stages in a 
particular time span. (2) The explicit decision purposes for which the evaluator 
has ostensibly been retained to provide information may "not correspond with the 
stage of development of the program. Thus, an evaluator may be asked to provide 
information appropriate to decisions in the formative stage while the program, 
in the completeness and specificity of its development, has not attained that stage. 



This type of situation occurs when an evaluator is asked to provide a formative 
evaluation in order to provide information for program modification when the 
program itself has not been developed to the extent that one would know what 
was an- anticipated part of the program and what would be a modification. Clearly, 
it is difficult to attend to the consideration of program modification when the 
nature of the program itself has not yet been well defined, yet this is a frequent 
occurrence . 

An evaluator must decide what to dc in such an instance. In our work 
we have followed the tack of forcing the completion of prior developmental stages 
before commencing activities towards the avowed purpose of the evaluation. 
Frequently this has meant that prior developmental stages [prior to the stage 
for which evaluation is requested) are approximated and parts of the various stages 
are completed simultaneously; these approximations become assumptions for the 
work that is to follow. Sometimes the penetrating questions of the evaluator, 
pinpointing the inadequacy of the prior development of the program, is sufficient 
feedback to stimulate the reconsideration, attention, and focus of the program 
on its appropriate developmental stage. 
The Prog ram 1 s Socio-political Setting 

An educational program exists in a socio-political setting which has 
defineable dimensions. The size and scope of the target program as well as .the 
nature of the community (ies) uniquely frame an evaluation effort. Indeed, the 
program itself can be placed within the political context of other school 
districts, lines of administrative responsibility and interacting programs. 

An organization's or program's socio-political setting represents an 
area of continued research and scholarly activity (Bind, 1970; Ferman, 1969; 
Gordon, 1973) which evaluators have all to frequently ignored. For example, 
Ferman has studied evaluation from the perspective of a system of social 



9 



interaction involving the evaluator, sponsor and program staff, while Bind has 
considered three problem areas that reflect, a program 1 s social setting. 

The Total Evaluation Effort 

In this presentation we have focused on the first step of the evaluation 
framing the decision context. If we were to develop this paper into a broader 
treatise on evaluation , similar attention and delineation would be directed to 
the evaluation steps that follow. The appropriate titles for some of these 
steps might be: Building the Evaluation System , Techn iques and Methodologies , 
and Evaluation Information and Reporting . In each of these areas old but still 
useful ideas on evaluation might be pulled together in an eclectic fashion to 
provide "new" insights into the nature of evaluation. 



10 



References 



Alkin, M.C. Evaluation theory development, Evaluation Comment, 1969 
, 2(1), 1-7. 

Alkin, M.C. A theory o£ evaluation. CSE Working Paper No. 18, 1971, CSE, UCLA. 

Bind, E. The impact of social setting upon evaluative research. In Evaluative 
Research. Pittsburgh: American Institutes for Research, 1970. ~~ 

Ferman, L. Some perspectives on evaluating social welfare programs. Annals 

of the American Acad emy of Political and Social Science , 1969, 335, 143-156. 

Glass, G. The growth of evaluation methodology. Unpublished manuscript 
Laboratory of Educational Research, University of Colorado, 1969. 

Gordon, C.W. The Study of educational evaluation in social context. Paper 
presented at the annual meeting of American Educational Research 
Association, New Orleans , February, 1973. 

Gouldner, A. Engineering and clinical approaches to consulting. In W. Bennis, 
K. Benne, $ R. Chin (Eds.) The planning of change: Readings in applied 
behavioral sciences , New York! Holt, Rinehart § Winston,~196T: 

Provus, M. Discrepancy evaluation: For educational prog ram improvement and 
assessment . Berkeley: McCutchan Publishing Corp. ,1971. 

Popham, W.J. An evaluation guidebook . Los Angeles: Instructional Obiectives 
Exchange, 1972. 

Scriven, M. Prose and cons about goal -free evaluation. Evaluation Comment 
1972, 3(4), 1-4. — — ' 

Stake, Robert. The countenance of educational evaluation. Teachers College 
Rerord , 1967, 68(7), 523-540. 

Stufflebeam, D.L. , Foley, W. , Guba, E., Hammond, R. , Merriman, H. $ Provus, M 
Educationa l evaluation and decision making . Itasca: Peacock Publishers, 
19 1 1 • 



ERIC 



: : ; 'J *6 . J .:'.':':: ';o o .:' • ..v.. 



.i: o:7 v:.jv; or opinions seated 
,:j; ; ici..I U. 5,0,1s, position 



ERLC 



